Japanese text segmentation : a comparison of different methods applied to Kanji

Written Japanese and Chinese contain no word delimiters such as spaces, so segmentation into words is the first step in processing text in these languages. Over the years several methods of segmentation that utilize various statistical and grammatical principles have been developed. I have implemented two such methods, the Tango algorithm by Ando and Lee and a hidden markov model by Papageorgiou. In this paper, I will contrast these two methods with an open-source knowledge-based parser Juman 5.1 in their ability to segment Japanese text.